Lecture 2 : Stochastic Multi - armed bandit ( IID model )

1.1 Reward model The next step in defining the model is to describe how the reward are generated. This is where the stochastic assumption and IID model comes in. In particular, we assume that the reward from each arm i follows a distribution νi with mean μi. When pulling an arm i, the reward will be generated independently from the distribution νi. More precisely, let Ht−1 denote the history until time t− 1 (including t− 1). We can write Ht−1 = {(I1, r1), . . . , (It−1, rt−1)}. Then, our assumption on the reward can be written as rt|(Ht−1, It = i) ∼ νi which also implies E[rt|Ht−1, It = i] = μi. In other words, given the history up to time t−1 and the choice of arm It, the reward is drawn independently with respect to the distribution of the chosen arm.